Learning Deep Architectures via Generalized Whitened Neural Networks
نویسنده
چکیده
Whitened Neural Network (WNN) is a recent advanced deep architecture, which improves convergence and generalization of canonical neural networks by whitening their internal hidden representation. However, the whitening transformation increases computation time. Unlike WNN that reduced runtime by performing whitening every thousand iterations, which degenerates convergence due to the ill conditioning, we present generalized WNN (GWNN), which has three appealing properties. First, GWNN is able to learn compact representation to reduce computations. Second, it enables whitening transformation to be performed in a short period, preserving good conditioning. Third, we propose a data-independent estimation of the covariance matrix to further improve computational efficiency. Extensive experiments on various datasets demonstrate the benefits of GWNN.
منابع مشابه
Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
We seek to improve deep neural networks by generalizing the pooling operations that play a central role in current architectures. We pursue a careful exploration of approaches to allow pooling to learn and to adapt to complex and variable patterns. The two primary directions lie in (1) learning a pooling function via (two strategies of) combining of max and average pooling, and (2) learning a p...
متن کاملDeep Learning in Multi-Layer Architectures of Dense Nuclei
In dense clusters of neurons in nuclei, cells may interconnect via soma-to-soma interactions, in addition to conventional synaptic connections. We illustrate this idea with a multi-layer architecture (MLA) composed of multiple clusters of recurrent sub-networks of spiking Random Neural Networks (RNN) with dense soma-to-soma interactions. We use this RNN-MLA architecture for deep learning. The i...
متن کاملWorkshop proposal: Deep Learning in Computational Cognitive Science
A new generation of deep neural network architectures has driven rapid advances in AI over the last ten years. These architectures include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and many variants and extensions. Computational cognitive scientists and neuroscientists have now begun to explore these techniques, and how they might combine with other computational t...
متن کاملImproved Explainability of Capsule Networks: Relevance Path by Agreement
Recent advancements in signal processing and machine learning domains have resulted in an extensive surge of interest in deep learning models due to their unprecedented performance and high accuracy for different and challenging problems of significant engineering importance. However, when such deep learning architectures are utilized for making critical decisions such as the ones that involve ...
متن کاملAnalysis of Deep Convolutional Neural Network Architectures
In computer vision many tasks are solved using machine learning. In the past few years, state of the art results in computer vision have been achieved using deep learning. Deeper machine learning architectures are better capable in handling complex recognition tasks, compared to previous more shallow models. Many architectures for computer vision make use of convolutional neural networks which ...
متن کامل